Hierarchical Category Classification Scheme Using Multiple Sets of Fully-Connected Networks With A CNN Based Integrated Circuit As Feature Extractor

ABSTRACT

CNN based integrated circuit is configured with a set of pre-trained filter coefficients or weights as a feature extractor of an input data. Multiple fully-connected networks (FCNs) are trained for use in a hierarchical category classification scheme. Each FCN is capable of classifying the input data via the extracted features in a specific level of the hierarchical category classification scheme. First, a root level FCN is used for classifying the input data among a set of top level categories. Then, a relevant next level FCN is used in conjunction with the same extracted features for further classifying the input data among a set of subcategories to the most probable category identified using the previous level FCN. Hierarchical category classification scheme continues for further detailed subcategories if desired.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part (CIP) to a co-pending U.S.patent application Ser. No. 15/709,220 for “Natural Language ProcessingUsing A CNN Based Integrated Circuit” filed on Sep. 19, 2017, which is aCIP to a co-pending U.S. patent application Ser. No. 15/694,711 for“Natural Language Processing Via A Two-dimensional Symbol HavingMultiple Ideograms Contained Therein” filed on Sep. 1, 2017, which is aCIP to a co-pending U.S. patent application Ser. No. 15/683,723 for“Two-dimensional Symbols For Facilitating Machine Learning Of CombinedMeaning Of Multiple Ideograms Contained Therein” filed on Aug. 22, 2017,which claims priority from a co-pending U.S. Provisional PatentApplication Ser. No. 62/541,081, entitled “Two-dimensional Symbol ForFacilitating Machine Learning Of Natural Languages Having LogosyllabicCharacters” filed on Aug. 3, 2017.

This application is a continuation-in-part (CIP) to a co-pending U.S.patent application Ser. No. 15/289,726 for “Digital Integrated CircuitFor Extracting Features Out Of An Input Image Based On Cellular NeuralNetworks” filed on Oct. 10, 2016.

This application claims priority from a co-pending U.S. ProvisionalPatent Application Ser. No. 62/548,905, entitled “Hierarchical CategoryClassification Scheme Using Multiple Sets of Fully-Connected NetworksWith A CNN Based Integrated Circuit As Feature Extractor” filed on Nov.12, 2017.

All of which are hereby incorporated by reference in their entirety forall purposes.

FIELD

The invention generally relates to the field of integrated circuit andmore particularly to a digital integrated circuit for extractingfeatures out of an input image based on Cellular Neural Networks.

BACKGROUND

Cellular Neural Networks or Cellular Nonlinear Networks (CNN) have beenapplied to many different fields and problems including, but limited to,image processing since 1988. However, most of the prior art CNNapproaches are either based on software solutions (e.g., ConvolutionalNeural Networks, Recurrent Neural Networks, etc.) or based on hardwarethat are designed for other purposes (e.g., graphic processing, generalcomputation, etc.). As a result, CNN prior approaches are too slow interm of computational speed and/or too expensive thereby impractical forprocessing large amount of imagery data. The imagery data can be fromany two-dimensional data (e.g., still photo, picture, a frame of a videostream, converted form of voice data, etc.)

SUMMARY

This section is for the purpose of summarizing some aspects of theinvention and to briefly introduce some preferred embodiments.Simplifications or omissions in this section as well as in the abstractand the title herein may be made to avoid obscuring the purpose of thesection. Such simplifications or omissions are not intended to limit thescope of the invention.

A hierarchical category classification scheme in conjunction with apre-configured CNN based integrated circuit is disclosed. According toone aspect of the invention, CNN based integrated circuit is configuredwith a set of pre-trained filter coefficients or weights as a featureextractor of an input data. Multiple fully-connected networks (FCNs) aretrained for use in a hierarchical category classification scheme. EachFCN is capable of classifying the input data via the extracted featuresin a specific level of the hierarchical category classification scheme.First, a root level FCN is used for classifying the input data among aset of top level categories. Then, a relevant next level FCN is used inconjunction with the same extracted features for further classifying theinput data among a set of subcategories to the most probable categoryidentified using the previous level FCN. Hierarchical categoryclassification scheme continues for further detailed subcategories ifdesired.

According to another aspect, improvement to computational speed offully-connected networks is to remove neurons or cells of subsequentlayers of a FCN that do not contribute to classification of thecategories. One of the techniques is to identify those neuron or cellshaving input value of zero for more than a threshold percentage of alldata in a desired training dataset in the particular domain.

According to yet another aspect, improvement to computational speed offully-connected networks is to remove any unnecessary connectionsbetween neurons or cells of subsequent layers of a FCN. One of thetechniques is to identify those connections with the numerical value ofcorresponding weights or coefficients equal to zero or less than apredefined value,

One advantage of the invention is to simplify the training,convolutional layers are pre-trained and stay the same for eachdifferent FCN. Only layers in each FCN need to be trained instead ofentire model. This saves time and labeled dataset size for training asonly relatively few layers need to be adjusted.

Another advantage of the invention is to be able to classify moredetailed categories due to hierarchical category classification scheme,because relatively few categories are used in each level of thehierarchical category classification scheme.

Objects, features, and advantages of the invention will become apparentupon examining the following detailed description of an embodimentthereof, taken in conjunction with the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the invention willbe better understood with regard to the following description, appendedclaims, and accompanying drawings as follows:

FIGS. 1A-1B are block diagrams illustrating an example integratedcircuit designed for extracting features from input imagery data inaccordance with one embodiment of the invention;

FIG. 2 is a function block diagram showing an example controllerconfigured for controlling operations of one or more CNN processingengines according to an embodiment of the invention;

FIG. 3 is a diagram showing an example CNN processing engine inaccordance with one embodiment of the invention;

FIG. 4 is a diagram showing M×M pixel locations within a (M+2)-pixel by(M+2)-pixel region, according to an embodiment of the invention;

FIGS. 5A-5C are diagrams showing three example pixel locations,according to an embodiment of the invention;

FIG. 6 is a diagram illustrating an example data arrangement forperforming 3×3 convolutions at a pixel location, according to oneembodiment of the invention;

FIG. 7 is a function block diagram illustrating an example circuitry forperforming 3×3 convolutions at a pixel location, according to oneembodiment of the invention;

FIG. 8 is a diagram showing an example rectification according to anembodiment of the invention;

FIGS. 9A-9B are diagrams showing two example 2×2 pooling operationsaccording to an embodiment of the invention;

FIG. 10 is a diagram illustrating a 2×2 pooling operation reducesM-pixel by M-pixel block to a (M/2)-pixel by (M/2)-pixel block inaccordance with one embodiment of the invention;

FIGS. 11A-11C are diagrams illustrating examples of M-pixel by M-pixelblocks and corresponding (M+2)-pixel by (M+2)-pixel region in an inputimage, according to one embodiment of the invention;

FIG. 12 is a diagram illustrating an example of a first set of memorybuffers for storing received imagery data in accordance with anembodiment of the invention;

FIG. 13A is a diagram showing two operational modes of an example secondset of memory buffers for storing filter coefficients in accordance withan embodiment of the invention;

FIG. 13B is a diagram showing example storage schemes of filtercoefficients in the second set of memory buffers, according to anembodiment of the invention;

FIG. 14 is a diagram showing a plurality of CNN processing enginesconnected as a loop via an example clock-skew circuit in accordance ofan embodiment of the invention;

FIG. 15 is a schematic diagram showing an example image processingtechnique based on convolutional neural networks in accordance with anembodiment of the invention;

FIG. 16 is a flowchart illustrating an example process of achieving atrained convolutional neural networks model having bi-valued 3×3 filterkernels in accordance with an embodiment of the invention;

FIG. 17 is a diagram showing an example filter kernel conversion schemein accordance with the invention;

FIG. 18 is a diagram showing an example data conversion scheme;

FIG. 19A is a schematic diagram showing data flow of an example featureextractor in accordance with one embodiment of the invention;

FIGS. 19B-19D are schematic diagrams showing an example hierarchicalclassification scheme with multiple fully-connected networks inaccordance with one embodiment of the invention;

FIG. 20 is a schematic diagram showing example connection patterns anexample FCN in accordance with one embodiment of the invention;

FIGS. 21A-21B are schematic diagrams showing a first example computationreduction scheme in FCN in accordance with one embodiment of theinvention;

FIGS. 22A-22B are schematic diagrams showing a second examplecomputation reduction scheme in FCN in accordance with one embodiment ofthe invention; and

FIG. 23 is a flowchart showing an example process of computationreduction scheme in a FCN according to an embodiment of the invention.

DETAILED DESCRIPTIONS

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the invention. However, itwill become obvious to those skilled in the art that the invention maybe practiced without these specific details. The descriptions andrepresentations herein are the common means used by those experienced orskilled in the art to most effectively convey the substance of theirwork to others skilled in the art. In other instances, well-knownmethods, procedures, and components have not been described in detail toavoid unnecessarily obscuring aspects of the invention.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment can be included in at least one embodiment of theinvention. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment, nor are separate or alternative embodiments mutuallyexclusive of other embodiments. Further, the order of blocks in processflowcharts or diagrams or circuits representing one or more embodimentsof the invention do not inherently indicate any particular order norimply any limitations in the invention. Used herein, the terms “top”,“bottom”, “right” and “left” are intended to provide relative positionsfor the purposes of description, and are not intended to designate anabsolute frame of reference

Embodiments of the invention are discussed herein with reference toFIGS. 1A-23. However, those skilled in the art will readily appreciatethat the detailed description given herein with respect to these figuresis for explanatory purposes as the invention extends beyond theselimited embodiments.

Referring first to FIG. 1A, it is shown a block diagram illustrating anexample digital integrated circuit (IC) 100 for extracting features outof an input image in accordance with one embodiment of the invention.

The integrated circuit 100 is implemented as a digital semi-conductorchip and contains a CNN processing engine controller 110, and one ormore neural networks (CNN) processing engines 102 operatively coupled toat least one input/output (I/O) data bus 120. Controller 110 isconfigured to control various operations of the CNN processing engines102 for extracting features out of an input image based on an imageprocessing technique by performing multiple layers of 3×3 convolutionswith rectifications or other nonlinear operations (e.g., sigmoidfunction), and 2×2 pooling operations. To perform 3×3 convolutionsrequires imagery data in digital form and corresponding filtercoefficients, which are supplied to the CNN processing engine 102 viainput/output data bus 120. It is well known that digital semi-conductorchip contains logic gates, multiplexers, register files, memories, statemachines, etc.

According to one embodiment, the digital integrated circuit 100 isextendable and scalable. For example, multiple copy of the digitalintegrated circuit 100 can be implemented on one semiconductor chip.

All of the CNN processing engines are identical. For illustratingsimplicity, only few (i.e., CNN processing engines 122 a-122 h, 132a-132 h) are shown in FIG. 1B. The invention sets no limit to the numberof CNN processing engines on a digital semi-conductor chip.

Each CNN processing engine 122 a-122 h, 132 a-132 h contains a CNNprocessing block 124, a first set of memory buffers 126 and a second setof memory buffers 128. The first set of memory buffers 126 is configuredfor receiving imagery data and for supplying the already receivedimagery data to the CNN processing block 124. The second set of memorybuffers 128 is configured for storing filter coefficients and forsupplying the already received filter coefficients to the CNN processingblock 124. In general, the number of CNN processing engines on a chip is2^(n), where n is an integer (i.e., 0, 1, 2, 3, . . . ). As shown inFIG. 1B, CNN processing engines 122 a-122 h are operatively coupled to afirst input/output data bus 130 a while CNN processing engines 132 a-132h are operatively coupled to a second input/output data bus 130 b. Eachinput/output data bus 130 a-130 b is configured for independentlytransmitting data (i.e., imagery data and filter coefficients). In oneembodiment, the first and the second sets of memory buffers compriserandom access memory (RAM). Each of the first and the second sets arelogically defined. In other words, respective sizes of the first and thesecond sets can be reconfigured to accommodate respective amounts ofimagery data and filter coefficients.

The first and the second I/O data bus 130 a-130 b are shown here toconnect the CNN processing engines 122 a-122 h, 132 a-132 h in asequential scheme. In another embodiment, the at least one I/O data busmay have different connection scheme to the CNN processing engines toaccomplish the same purpose of parallel data input and output forimproving performance.

FIG. 2 is a diagram showing an example controller 200 for controllingvarious operations of at least one CNN processing engine configured onthe integrated circuit. Controller 200 comprises circuitry to controlimagery data loading control 212, filter coefficients loading control214, imagery data output control 216, and image processing operationscontrol 218. Controller 200 further includes register files 220 forstoring the specific configuration (e.g., number of CNN processingengines, number of input/output data bus, etc.) in the integratedcircuit.

Image data loading control 212 controls loading of imagery data torespective CNN processing engines via the corresponding I/O data bus.Filter coefficients loading control 214 controls loading of filtercoefficients to respective CNN processing engines via corresponding I/Odata bus. Imagery data output control 216 controls output of the imagerydata from respective CNN processing engines via corresponding I/O databus. Image processing operations control 218 controls various operationssuch as convolutions, rectifications and pooling operations which can bedefined by user of the integrated circuit via a set of user defineddirectives (e.g., file contains a series of operations such asconvolution, rectification, pooling, etc.).

More details of a CNN processing engine 302 are shown in FIG. 3. A CNNprocessing block 304 contains digital circuitry that simultaneouslyobtains M×M convolution operations results by performing 3×3convolutions at M×M pixel locations using imagery data of a (M+2)-pixelby (M+2)-pixel region and corresponding filter coefficients from therespective memory buffers. The (M+2)-pixel by (M+2)-pixel region isformed with the M×M pixel locations as an M-pixel by M-pixel centralportion plus a one-pixel border surrounding the central portion. M is apositive integer. In one embodiment, M equals to 14 and therefore, (M+2)equals to 16, M×M equals to 14×14=196, and M/2 equals 7.

FIG. 4 is a diagram showing a diagram representing (M+2)-pixel by(M+2)-pixel region 410 with a central portion of M×M pixel locations 420used in the CNN processing engine 302.

Imagery data may represent characteristics of a pixel in the input image(e.g., one of the color (e.g., RGB (red, green, blue)) values of thepixel, or distance between pixel and observing location). Generally, thevalue of the RGB is an integer between 0 and 255. Values of filtercoefficients are floating point integer numbers that can be eitherpositive or negative.

In order to achieve faster computations, few computational performanceimprovement techniques have been used and implemented in the CNNprocessing block 304. In one embodiment, representation of imagery datauses as few bits as practical (e.g., 5-bit representation). In anotherembodiment, each filter coefficient is represented as an integer with aradix point. Similarly, the integer representing the filter coefficientuses as few bits as practical (e.g., 12-bit representation). As aresult, 3×3 convolutions can then be performed using fixed-pointarithmetic for faster computations.

Each 3×3 convolution produces one convolution operations result, Out(m,n), based on the following formula:

$\begin{matrix}{{{Out}\left( {m,n} \right)} = {{\sum\limits_{{1 \leq i},{j \leq 3}}\; {{{In}\left( {m,n,i,j} \right)} \times {C\left( {i,j} \right)}}} - b}} & (1)\end{matrix}$

where:

-   -   m, n are corresponding row and column numbers for identifying        which imagery data (pixel) within the (M+2)-pixel by (M+2)-pixel        region the convolution is performed;    -   In(m,n,i,j) is a 3-pixel by 3-pixel area centered at pixel        location (m, n) within the region;    -   C(i, j) represents one of the nine weight coefficients C(3×3),        each corresponds to one of the 3-pixel by 3-pixel area;    -   b represents an offset coefficient; and    -   i, j are indices of weight coefficients C(i, j).

Each CNN processing block 304 produces M×M convolution operationsresults simultaneously and, all CNN processing engines performsimultaneous operations.

FIGS. 5A-5C show three different examples of the M×M pixel locations.The first pixel location 531 shown in FIG. 5A is in the center of a3-pixel by 3-pixel area within the (M+2)-pixel by (M+2)-pixel region atthe upper left corner. The second pixel location 532 shown in FIG. 5B isone pixel data shift to the right of the first pixel location 531. Thethird pixel location 533 shown in FIG. 5C is a typical example pixellocation. M×M pixel locations contains multiple overlapping 3-pixel by3-pixel areas within the (M+2)-pixel by (M+2)-pixel region.

To perform 3×3 convolutions at each sampling location, an example dataarrangement is shown in FIG. 6. Imagery data (i.e., In(3×3)) and filtercoefficients (i.e., weight coefficients C(3×3) and an offset coefficientb) are fed into an example CNN 3×3 circuitry 600. After 3×3 convolutionsoperation in accordance with Formula (1), one output result (i.e.,Out(1×1)) is produced. At each sampling location, the imagery dataIn(3×3) is centered at pixel coordinates (m, n) 605 with eight immediateneighbor pixels 601-604, 606-609.

FIG. 7 is a function diagram showing an example CNN 3×3 circuitry 700for performing 3×3 convolutions at each pixel location. The circuitry700 contains at least adder 721, multiplier 722, shifter 723, rectifier724 and pooling operator 725. In a digital semi-conductorimplementation, all of these can be achieved with logic gates andmultiplexers, which are generated using well-known methods (e.g.,hardware description language such as Verilog, etc.). Adder 721 andmultiplier 722 are used for addition and multiplication operations.Shifter 723 is for shifting the output result in accordance withfixed-point arithmetic involved in the 3×3 convolutions. Rectifier 724is for setting negative output results to zero. Pooling operator 725 isfor performing 2×2 pooling operations.

Imagery data are stored in a first set of memory buffers 306, whilefilter coefficients are stored in a second set of memory buffers 308.Both imagery data and filter coefficients are fed to the CNN block 304at each clock of the digital integrated circuit. Filter coefficients(i.e., C(3×3) and b) are fed into the CNN processing block 304 directlyfrom the second set of memory buffers 308. However, imagery data are fedinto the CNN processing block 304 via a multiplexer MUX 305 from thefirst set of memory buffers 306. Multiplexer 305 selects imagery datafrom the first set of memory buffers based on a clock signal (e.g.,pulse 312).

Otherwise, multiplexer MUX 305 selects imagery data from a firstneighbor CNN processing engine (from the left side of FIG. 3 not shown)through a clock-skew circuit 320.

At the same time, a copy of the imagery data fed into the CNN processingblock 304 is sent to a second neighbor CNN processing engine (to theright side of FIG. 3 not shown) via the clock-skew circuit 320.Clock-skew circuit 320 can be achieved with known techniques (e.g., a Dflip-flop 322).

The first neighbor CNN processing engine may be referred to as anupstream neighbor CNN processing engine in the loop formed by theclock-skew circuit 320. The second neighbor CNN processing engine may bereferred to as a downstream CNN processing engine. In anotherembodiment, when the data flow direction of the clock-skew circuit isreversed, the first and the second CNN processing engines are alsoreversed becoming downstream and upstream neighbors, respectively.

After 3×3 convolutions for each group of imagery data are performed forpredefined number of filter coefficients, convolution operations resultsOut(m, n) are sent to the first set of memory buffers via anothermultiplex MUX 307 based on another clock signal (e.g., pulse 311). Anexample clock cycle 310 is drawn for demonstrating the time relationshipbetween pulse 311 and pulse 312. As shown pulse 311 is one clock beforepulse 312, as a result, the 3×3 convolution operations results arestored into the first set of memory buffers after a particular block ofimagery data has been processed by all CNN processing engines throughthe clock-skew circuit 320.

After the convolution operations result Out(m, n) is obtained fromFormula (1), rectification procedure may be performed as directed byimage processing control 218. Any convolution operations result, Out(m,n), less than zero (i.e., negative value) is set to zero. In otherwords, only positive value of output results are kept. FIG. 8 shows twoexample outcomes of rectification. A positive output value 10.5 retainsas 10.5 while −2.3 becomes 0. Rectification causes non-linearity in theintegrated circuits.

If a 2×2 pooling operation is required, the M×M output results arereduced to (M/2)×(M/2). In order to store the (M/2)×(M/2) output resultsin corresponding locations in the first set of memory buffers,additional bookkeeping techniques are required to track proper memoryaddresses such that four (M/2)×(M/2) output results can be processed inone CNN processing engine.

To demonstrate a 2×2 pooling operation, FIG. 9A is a diagram graphicallyshowing first example output results of a 2-pixel by 2-pixel block beingreduced to a single value 10.5, which is the largest value of the fouroutput results. The technique shown in FIG. 9A is referred to as “maxpooling”. When the average value 4.6 of the four output results is usedfor the single value shown in FIG. 9B, it is referred to as “averagepooling”. There are other pooling operations, for example, “mixed maxaverage pooling” which is a combination of “max pooling” and “averagepooling”. The main goal of the pooling operation is to reduce size ofthe imagery data being processed. FIG. 10 is a diagram illustrating M×Mpixel locations, through a 2×2 pooling operation, are reduced to(M/2)×(M/2) locations, which is one fourth of the original size.

An input image generally contains a large amount of imagery data. Inorder to perform image processing operations. The input image 1100 ispartitioned into M-pixel by M-pixel blocks 1111-1112 as shown in FIG.11A. Imagery data associated with each of these M-pixel by M-pixelblocks is then fed into respective CNN processing engines. At each ofthe M×M pixel locations in a particular M-pixel by M-pixel block, 3×3convolutions are simultaneously performed in the corresponding CNNprocessing block.

Although the invention does not require specific characteristicdimension of an input image, the input image may be required to resizeto fit to a predefined characteristic dimension for certain imageprocessing procedures. In an embodiment, a square shape with(2^(K)×M)-pixel by (2^(K)×M)-pixel is required. K is a positive integer(e.g., 1, 2, 3, 4, etc.). When M equals 14 and K equals 4, thecharacteristic dimension is 224. In another embodiment, the input imageis a rectangular shape with dimensions of (2^(I)×M)-pixel and(2^(J)×M)-pixel, where I and J are positive integers.

In order to properly perform 3×3 convolutions at pixel locations aroundthe border of a M-pixel by M-pixel block, additional imagery data fromneighboring blocks are required. FIG. 11B shows a typical M-pixel byM-pixel block 1120 (bordered with dotted lines) within a (M+2)-pixel by(M+2)-pixel region 1130. The (M+2)-pixel by (M+2)-pixel region is formedby a central portion of M-pixel by M-pixel from the current block, andfour edges (i.e., top, right, bottom and left) and four corners (i.e.,top-left, top-right, bottom-right and bottom-left) from correspondingneighboring blocks. Additional details are shown in FIG. 12 andcorresponding descriptions for the first set of memory buffers.

FIG. 11C shows two example M-pixel by M-pixel blocks 1122-1124 andrespective associated (M+2)-pixel by (M+2)-pixel regions 1132-1134.These two example blocks 1122-1124 are located along the perimeter ofthe input image. The first example M-pixel by M-pixel block 1122 islocated at top-left corner, therefore, the first example block 1122 hasneighbors for two edges and one corner. Value “0” s are used for the twoedges and three corners without neighbors (shown as shaded area) in theassociated (M+2)-pixel by (M+2)-pixel region 1132 for forming imagerydata. Similarly, the associated (M+2)-pixel by (M+2)-pixel region 1134of the second example block 1124 requires “0”s be used for the top edgeand two top corners. Other blocks along the perimeter of the input imageare treated similarly. In other words, for the purpose to perform 3×3convolutions at each pixel of the input image, a layer of zeros (“0”s)is added outside of the perimeter of the input image. This can beachieved with many well-known techniques. For example, default values ofthe first set of memory buffers are set to zero. If no imagery data isfilled in from the neighboring blocks, those edges and corners wouldcontain zeros.

Furthermore, an input image can contain a large amount of imagery data,which may not be able to be fed into the CNN processing engines in itsentirety. Therefore, the first set of memory buffers is configured onthe respective CNN processing engines for storing a portion of theimagery data of the input image. The first set of memory bufferscontains nine different data buffers graphically illustrated in FIG. 12.Nine buffers are designed to match the (M+2)-pixel by (M+2)-pixel regionas follows:

1) buffer-0 for storing M×M pixels of imagery data representing thecentral portion;2) buffer-1 for storing 1×M pixels of imagery data representing the topedge;3) buffer-2 for storing M×1 pixels of imagery data representing theright edge;4) buffer-3 for storing 1×M pixels of imagery data representing thebottom edge;5) buffer-4 for storing M×1 pixels of imagery data representing the leftedge;6) buffer-5 for storing 1×1 pixels of imagery data representing the topleft corner;7) buffer-6 for storing 1×1 pixels of imagery data representing the topright corner;8) buffer-7 for storing 1×1 pixels of imagery data representing thebottom right corner; and9) buffer-8 for storing 1×1 pixels of imagery data representing thebottom left corner.

Imagery data received from the I/O data bus are in form of M×M pixels ofimagery data in consecutive blocks. Each M×M pixels of imagery data isstored into buffer-0 of the current block. The left column of thereceived M×M pixels of imagery data is stored into buffer-2 of previousblock, while the right column of the received M×M pixels of imagery datais stored into buffer-4 of next block. The top and the bottom rows andfour corners of the received M×M pixels of imagery data are stored intorespective buffers of corresponding blocks based on the geometry of theinput image (e.g., FIGS. 11A-11C).

An example second set of memory buffers for storing filter coefficientsare shown in FIG. 13A. In one embodiment, a pair of independent buffersBuffer0 1301 and Buffer1 1302 is provided. The pair of independentbuffers allow one of the buffers 1301-1302 to receive data from the I/Odata bus 1330 while the other one to feed data into a CNN processingblock (not shown). Two operational modes are shown herein.

Example storage schemes of filter coefficients are shown in FIG. 13B.Each of the pair of buffers (i.e., Buffer0 1301 or Buffer1 1302) has awidth (i.e., word size 1310). In one embodiment, the word size is120-bit. Accordingly, each of the filter coefficients (i.e., C(3×3) andb) occupies 12-bit in the first example storage scheme 1311. In thesecond example storage scheme 1312, each filter coefficient occupies6-bit thereby 20 coefficients are stored in each word. In the thirdexample scheme 1313, 3-bit is used for each coefficient hence four setsof filter coefficients (40 coefficients) are stored. Finally, in thefourth example storage scheme 1314, 80 coefficients are stored in eachword, each coefficient occupies 1.5-bit.

In another embodiment, a third memory buffer can be set up for storingentire filter coefficients to avoid I/O delay. In general, the inputimage must be at certain size such that all filter coefficients can bestored. This can be done by allocating some unused capacity in the firstset of memory buffers to accommodate such a third memory buffer. Sinceall memory buffers are logically defined in RAM (Random-Access Memory),well known techniques may be used for creating the third memory buffer.In other words, the first and the second sets of memory buffers can beadjusted to fit different amounts of imagery data and/or filtercoefficients. Furthermore, the total amount of RAM is dependent uponwhat is required in image processing operations.

When more than one CNN processing engine is configured on the integratedcircuit. The CNN processing engine is connected to first and secondneighbor CNN processing engines via a clock-skew circuit. Forillustration simplicity, only CNN processing block and memory buffersfor imagery data are shown. An example clock-skew circuit 1440 for agroup of CNN processing engines are shown in FIG. 14. The CNN processingengines connected via the second example clock-skew circuit 1440 form aloop. In other words, each CNN processing engine sends its own imagerydata to a first neighbor and, at the same time, receives a secondneighbor's imagery data. Clock-skew circuit 1440 can be achieved withwell-known manners. For example, each CNN processing engine is connectedwith a D flip-flop 1442.

A special case with only two CNN processing engines are connected in aloop, the first neighbor and the second neighbor are the same.

Referring now to FIG. 15, it is a schematic diagram showing an exampleimage processing technique based on convolutional neural networks inaccordance with an embodiment of the invention. Based on convolutionalneural networks, multi-layer input imagery data 1511 a-1511 c isprocessed with convolutions using a first set of filters or weights1520. Since the imagery data 1511 a-1511 c is larger than the filters1520. Each corresponding overlapped sub-region 1515 of the imagery datais processed. After the convolutional results are obtained, activationmay be conducted before a first pooling operation 1530. In oneembodiment, activation is achieved with rectification performed in arectified linear unit (ReLU). As a result of the first pooling operation1530, the imagery data is reduced to a reduced set of imagery data 1531a-1531 c. For 2×2 pooling, the reduced set of imagery data is reduced bya factor of 4 from the previous set.

The previous convolution-to-pooling procedure is repeated. The reducedset of imagery data 1531 a-1531 c is then processed with convolutionsusing a second set of filters 1540. Similarly, each overlappedsub-region 1535 is processed. Another activation can be conducted beforea second pooling operation 1540. The convolution-to-pooling proceduresare repeated for several layers and finally connected to aFully-connected Networks (FCN) 1560. In image classification, respectiveprobabilities of predefined categories can be computed in FCN 1560.

This repeated convolution-to-pooling procedure is trained using a knowndataset or database. For image classification, the dataset contains thepredefined categories. A particular set of filters, activation andpooling can be tuned and obtained before use for classifying an imagerydata, for example, a specific combination of filter types, number offilters, order of filters, pooling types, and/or when to performactivation. In one embodiment, convolutional neural networks are basedon Visual Geometry Group (VGG16) architecture neural nets, whichcontains 13 convolutional layers and three fully-connected networklayers.

A trained convolutional neural networks model is achieved with anexample set of operations 1600 shown in FIG. 16. At action 1602, aconvolutional neural networks model is first obtained by training theconvolutional neural networks model based on image classification of alabeled dataset, which contains a sufficiently large number of inputdata (e.g., imagery data, converted voice data, optical characterreorganization (OCR) data, etc.). For example, there are at least 4,000data for each category. In other words, each data in the labeled datasetis associated with a category to be classified. The convolutional neuralnetworks model includes multiple ordered filter groups (e.g., eachfilter group corresponds to a convolutional layer in the convolutionalneural networks model). Each filter in the multiple ordered filtergroups contains a standard 3×3 filter kernel (i.e., nine coefficients infloating point number format (e.g., standard 3×3 filter kernel 1710 inFIG. 17)). Each of the nine coefficients can be any negative or positivereal number (i.e., a number with fraction). The initial convolutionalneural networks model may be obtained from many different frameworksincluding, but not limited to, Mxnet, caffe, tensorflow, etc.

Then, at action 1604, the convolutional neural networks model ismodified by converting respective standard 3×3 filter kernels 1710 tocorresponding bi-valued 3×3 filter kernels 1720 of a currently-processedfilter group in the multiple ordered filter groups based on a set ofkernel conversion schemes. In one embodiment, each of the ninecoefficients C(i,j) in the corresponding bi-valued 3×3 filter kernel1720 is assigned a value ‘A’ equal to the average of absolutecoefficient values multiplied by the sign of corresponding coefficientsin the standard 3×3 filter kernel 1710 shown in following formula:

$\begin{matrix}{A = {\sum\limits_{{1 \leq i},{j \leq 3}}{{{C\left( {i,j} \right)}}/9}}} & (2)\end{matrix}$

Filter groups are converted one at a time in the order defined in themultiple ordered filter groups. In certain situation, two consecutivefilter groups are optionally combined such that the training of theconvolutional neural networks model is more efficient.

Next, at action 1606, the modified convolutional neural networks modelis retrained until a desired convergence criterion is met or achieved.There are a number of well known convergence criteria including, but notlimited to, completing a predefined number of retraining operation,converging of accuracy loss due to filter kernel conversion, etc. In oneembodiment, all filter groups including already converted in previousretraining operations can be changed or altered for fine tuning. Inanother embodiment, the already converted filter groups are frozen orunaltered during the retraining operation of the currently-processedfilter group.

Process 1600 moves to decision 1608, it is determined whether there isanother unconverted filter group. If ‘yes’, process 1600 moves back torepeat actions 1604-1606 until all filter groups have been converted.Decision 1608 becomes ‘no’ thereafter. At action 1610, coefficients ofbi-valued 3×3 filter kernels in all filter groups are transformed from afloating point number format to a fixed point number format toaccommodate the data structure required in the CNN based integratedcircuit. Furthermore, the fixed point number is implemented asreconfigurable circuits in the CNN based integrated circuit. In oneembodiment, the coefficients are implemented using 12-bit fixed pointnumber format.

FIG. 18 is a diagram showing an example data conversion scheme forconverting data from 8-bit [0-255] to 5-bit [0-31] per pixel. Forexample, bits 0-7 becomes 0, bits 8-15 becomes 1, etc.

Referring now to FIG. 19A, it is shown data flow of an example featureextractor 1920 in accordance with one embodiment of the invention. Inputdata or imagery data 1910 are fed into a feature extractor 1920 toproduce a set of extracted features 1930. Example input data or imagerydata 1910 include, but are not limited to, still photo, picture, a frameof a video stream, converted form of voice data, optical characterrecognition (OCR), etc. Feature extractor 1920 is a CNN based integratedcircuit (e.g., CNN based IC 100 of FIGS. 1A-1B) with a set of filtercoefficients or weights configured thereon. The set of filtercoefficients or weights are pre-trained for a specific domain (e.g.,image, converted voice spectrum, etc.). Pre-trained filter coefficientsare loaded into the CNN based integrated circuit entirely or in-partdepending upon the amount and size of the filter coefficients. Extractedfeatures 1930 are results of performing a number of convolutionaloperations along with activation and pooling operations over the inputdata 1920 in the feature extractor 1920.

As described in process 1600 of FIG. 16, a convolutional neural networksmodel is trained for the CNN based integrated circuit. The entire set oftrained coefficients or weights are pre-configured to the CNN basedintegrated circuit as a feature extractor for a particular data format(e.g., imagery data, converted voice data, optical character recognition(OCR), etc.). In general, there are many convolutional layers with manyfilters in each layer. In one embodiment, VGG16 model contains 13convolutional layers. In a software based image classification task,computations for the convolutional layers take majority of computations(e.g., 90%) traditionally. This computations is drastically reduced witha dedicated hardware such as CNN based IC 100.

Input data is then classified in a predefined set of categories in ahierarchical category classification scheme using multiplefully-connected networks (FCNs), which are pre-trained for various levelof categories.

Each FCN is capable of classifying the input data via the extractedfeatures in a specific level of the hierarchical category classificationscheme. First, a root level FCN is used for classifying the input dataamong a set of top level categories. Then, a relevant next level FCN isused in conjunction with the same extracted features for furtherclassifying the input data among a set of subcategories to the categorydetermined in previous level. Hierarchical category classificationscheme continues for further detailed subcategories if desired

FIGS. 19B-19D are schematic diagrams showing an example hierarchicalclassification scheme with multiple fully-connected networks inaccordance with one embodiment of the invention.

First, extracted features 1930 are processed through a root level FCN(i.e., Model-L0 1940). The results are respective probabilities 1960 ofa set of top level categories 1950 in the first level of classificationof the hierarchical category classification scheme. As shown in thisexample in FIG. 19B, the top level categories are “Animal”, “Car”, andso on. The input data is found to have 90.2% probability to be Category“Animal”.

Next, the same extracted features 1930 are processed through a relevantnext level FCN (i.e., Model-L1-Animal 1941). The results are againrespective probabilities 1961 of a set of subcategories 1951 to Category“Animal”. In this example shown in FIG. 19C, subcategories are “Dog”,“Cat”, “Horse”, etc. In this level, the input data is found to have88.8% probability to be Category “Dog”.

Then, the same extracted features 1930 are processed through a anotherrelevant next level FCN (i.e., Model-L2-Dog 1942). Probabilities 1962 ofthe more detailed subcategories 1952 to Category “Dog” are resulted. Itcan be seen that the more detailed subcategories for “Dog” are “Poodle”,“Bulldog”, . . . , “Golden Retriever”. In this example shown in FIG.19D, the input image is found 98.5% probability to be “GoldenRetriever”.

Using hierarchical category classification scheme, each FCN isresponsible for classifying input data among a relatively smaller numberof categories thereby more efficient and accurate. And training of animage classification model becomes relatively easier because only fewlayers in FCN need to be adjusted.

FIG. 20 is a schematic diagram showing example connection patterns anexample FCN in accordance with one embodiment of the invention.Extracted features 2030 and a set of categories 2050 are fully-connectedin an example FCN, which is shown as more than one layers (e.g., FCNLayer-1 2040 a, FCN Layer-2 2040 b, etc.).

A weight or coefficient is associated with each connection between afeature in the extracted features 2030 and a cell or neuron of the FCNlayer-1 2040 a. Similarly a weight is associated with each and everyconnections between cells of the layers in FCN. For illustrationsimplicity and clarity, only few features, cells and categories aredrawn and shown in FIG. 20. In order to obtain respective probabilitiesof the set of categories, many computations are required as each valuein a neuron is multiplied with respective weights for each level of theFCN. As the feature extraction portion is sped up drastically with thededicated hardware (i.e., CNN based IC 100), the computations in FCNbecome more dominated. Techniques to computation speed in the FCNportion becomes important.

One of the techniques is to reduce the number of computations based onthe numerical value of weights obtained in training of a fully-connectednetworks. FIG. 21A is a schematic diagram showing connection patternsbetween FCN Layer-A 2121 and FCN Layer-B 2122. FCN Layer-A 2121 containsI cells or neurons while FCN Layer-B 2122 contains J cells or neurons.To fully connect these two layers, there are the total number ofconnections equal to I×J. Each connection is associated with a weight.For example, the first neuron or cell of FCN Layer-A 2121 connects tothe first neuron or cell of FCN Layer-B 2122 via a connection havingweight W₁₁. The second cell of FCN Layer-A 2121 to the first cell of FCNLayer-B 2122 is associated with weight W₂₁. For a generic connection,the corresponding weight W_(ij) with i in [1,I] and j in [1,J]. In oneembodiment, I and J are both large numbers, for example, I is largerthan 4096 while J is 4096. For layer connecting the feature extractorand FCN, I can be 25,088 and I is 4096.

To reduce the number of computations, the connection is removed when thenumerical value of the corresponding weight or coefficient is equal tozero or less than a predefined threshold value. FIG. 21B shows such anexample for W₁₁=0 and W_(2J) is less than a threshold value V_(th). Dueto huge number of connections existed between two FCN layers 2121-2122,only few representative connections are shown in FIG. 21B for simplicityand clarity. For those having ordinary skill in the art would understandfully-connected networks are too large to shown in a schematic diagram.

Another computation reduction scheme is to remove those neurons or cellsin a FCN layer that do not provide any contribution to the outputresults (i.e., those neurons or cells having numerical value of zero).FIG. 23 is a flowchart illustrating an example process 2300 of thecomputation reduction scheme in a FCN.

Process 2300 starts at action 2302 to obtain input values in each layerof a FCN by running all data in a desired dataset (e.g., labeled datasetfor training). Then, at action 2304, those neurons that have numericalvalue of zero at least certain threshold percentage of times areidentified. Finally, at action 2306, the identified neurons to reducethe size of the FCN layers are removed thereby compressing the FCNmodel. The threshold percentage can be empirically-determined with eachdifferent dataset, for example, 90% for a particular dataset in aspecific domain.

A second example computation reduction scheme is shown as schematicdiagrams in FIGS. 22A-22B. Neurons or cells of FCN Layer-C 2221 arefully-connected with neurons of cells of FCN Layer-D 2222 in FIG. 22A.Any neuron or cell has a numerical value of zero is checked to see if itis zero 2231-2232 for at least certain threshold percentage of times forall data in a desired dataset (i.e., training dataset for a particularcategory). If so, the corresponding neuron or cell is removed in the FCNas shown in FIG. 22B and the connections are reduced. As the number ofneurons or cells are huge numbers in each layer of a FCN, the reductionof these non-contributing neurons/cells can improve the computationalspeed and still maintain accuracy.

For better extracting features in different domains, like speech, facerecognition, gesture recognition and etc, different sets of configuredconvolution layer coefficients are provided for that domain. And theparticular set of convolution layers is used as a general featureextractor for the specific tasks in that domain. For example, thespecific task of family members face recognition in the domain of facerecognition, and the specific task of company employee face recognitionalso in the same domain of face recognition. And these two specifictasks can share the same set of convolution layers coefficients used forface detection but using different FCNs.

For different application domains, a set of pre-trained convolutioncoefficients based on large dataset is provided. For example, for imagedomain, a set of pre-trained ImageNet bi-value model is provided andloaded into the CNN based IC, and this convolution coefficients may beused in 1000-categories image related area. Based on that, users onlyuse fixed convolution coefficients to train FCNs only. For voice domain,a set of pre-trained convolution coefficients based on voice spectrum isprovided.

Although the invention has been described with reference to specificembodiments thereof, these embodiments are merely illustrative, and notrestrictive of, the invention. Various modifications or changes to thespecifically disclosed example embodiments will be suggested to personsskilled in the art. For example, whereas the input image has been shownand described as partitioning into M-pixel by M-pixel blocks in certainorder, other orders may be used in the invention to achieve the same,for example, the ordering of the M-pixel by M-pixel blocks may becolumn-wise instead of row-wise. Furthermore, whereas M-pixel by M-pixelblocks have been shown and described using M equals to 14 as an example.M can be chosen as other positive integers to accomplish the same, forexample, 16, 20, 30, etc. Additionally, whereas the 3×3 convolution and2×2 pooling have been shown and described, other types of convolutionand pooling operations may be used to achieve the same, for example, 5×5convolution and 3×3 pooling. Further, whereas hierarchical categoryclassification scheme has been shown and described in a three-levelhierarchy example, the invention does not set limit as to how manylevels can be employed in the scheme, for example, four, five or evenmore levels of hierarchy. In summary, the scope of the invention shouldnot be restricted to the specific example embodiments disclosed herein,and all modifications that are readily suggested to those of ordinaryskill in the art should be included within the spirit and purview ofthis application and scope of the appended claims.

What is claimed is:
 1. A system for classifying input data using ahierarchical category classification scheme comprising: a cellularneural networks (CNN) based integrated circuit being loaded withpre-trained filter coefficients of convolutional layers for extractingfeatures out of an input data that belong to a particular domain; and amulti-processor computing unit configured for using a plurality ofpre-trained fully-connected networks (FCNs) in a hierarchical categoryclassification scheme to identify the input data as a most probablecategory based on the features extracted from the CNN based integratedcircuit.
 2. The system of claim 1, wherein the CNN based integratedcircuit comprises a plurality of cellular neural networks (CNN)processing engines operatively coupled to the at least one input/outputdata bus, the plurality of CNN processing engines being connected in aloop with a clock-skew circuit, each CNN processing engine comprising: aCNN processing block configured for simultaneously obtaining convolutionoperations results using the input data and the pre-trained filtercoefficients; a first set of memory buffers operatively coupling to theCNN processing block for storing the input data to be fed into the CNNblock; and a second set of memory buffers operative coupling to the CNNprocessing block for storing the pre-trained filter coefficients to befed into the CNN processing block.
 3. The system of claim 1, wherein theparticular domain comprises converted voice spectrum data.
 4. The systemof claim 1, wherein the particular domain comprises imagery data forface recognition.
 5. The system of claim 1, wherein the particulardomain comprises imagery data for optical character recognition.
 6. Thesystem of claim 1, wherein the particular domain comprises convertedvoice spectrum data for voice recognition.
 7. The system of claim 1,wherein the plurality of pre-trained FCNs comprises at least a rootlevel fully-connected network (FCN) and a group of next level FCNs. 8.The system of claim 7, wherein the root level FCN is used forclassifying the input data by identify the input data as the mostprobable category among a set of top level categories based on thefeatures extracted from the CNN based integrated circuit.
 9. The systemof claim 7, wherein a relevant one of the next level FCNs is used forclassifying the input data among a set of subcategories to the mostprobable category identified using previous level FCN based on thefeatures extracted from the CNN based integrated circuit.
 10. The systemof claim 1, wherein each of the plurality of FCNs contains at least oneFCN layers with each FCN layer containing multiple neurons or cells. 11.The system of claim 10, wherein said each FCN layer is reduced in sizeby removing those neurons or cells have input value of zero for morethan a threshold percentage of all data in a desired dataset.
 12. Thesystem of claim 11, wherein the desired dataset comprises a labeleddataset for training in the particular domain.
 13. The system of claim11, wherein the threshold percentage is determined from the desiredtraining dataset.
 14. The system of claim 11, wherein the thresholdpercentage is 90%.
 15. The system of claim 10, wherein said each FCNlayer is reduced in size by removing those connections when numericalvalue of corresponding weights is equal to zero.
 16. The system of claim10, wherein said each FCN layer is reduced in size by removing thoseconnections when numerical value of corresponding weights is less than athreshold value.
 17. The system of claim 16, wherein the threshold valueis determined that the contribution of the connection is too small forcorresponding feature represented by the neuron.
 18. The system of claim1, wherein the pre-trained filter coefficients of convolutional layerscomprises bi-valued 3×3 filter coefficients or weights.
 19. The systemof claim 1, wherein the hierarchical category classification scheme isderived from Visual Geometry Group's VGG16 model with 13 convolutionallayers and three FCN layers.